{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 如何获得训练模型需要的数据集?"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们需要什么样的数据集呢？\n",
    "- 欺诈相关：与经济欺诈有关的内容主题。\n",
    "- 文本形式：会议场景的欺诈分析主要针对参会人的发言内容进行文本分析。\n",
    "- 对话格式：会议场景的诈骗是通过人与人之间的对话发生的，对话格式要比事后报道更贴合。\n",
    "- 语言：最好是中文\n",
    "\n",
    "因此，需要先查找可用的数据集。\n",
    "\n",
    "> 总结：确定数据集`类型`，"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 查找数据集的途径\n",
    "1. 去HuggingFace以及modelscope等开源模型网站上查找是否有相关领域的数据集。\n",
    "2. 去一些防诈骗网站（例如：反诈中心）上找公开介绍的诈骗案例，或者可直接下载的数据集。\n",
    "3. 让chatgpt来生成诈骗话术。\n",
    "\n",
    "### 综合分析\n",
    "1. 几种途径均没有直接可用的数据集，但有一个`想法`或许可行：将`案例报道`和`GPT生成`两者结合起来，\n",
    "2. 即以Fraud_New_Reports的诈骗案例为内容背景，调用GPT生成我们需要的对话数据集。\n",
    "3. 就是将上面表格截图中的F列内容作为上下文，让GPT以此上下文作为背景，生成此故事所描述的对话集，\n",
    "4. 并打上分类标签加以训练，理论上能实现我们想要的目标，但需要实际验证下。"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 原始案例集是一个csv格式，我们使用`pandas`提供的`read_csv函数`来加载整个表格的数据，使用head函数来预览数据集的前几条内容。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>标题</th>\n",
       "      <th>标题链接</th>\n",
       "      <th>首页缩略封面图</th>\n",
       "      <th>首页缩略文段</th>\n",
       "      <th>发布者</th>\n",
       "      <th>文章正文内容</th>\n",
       "      <th>图片链接</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>近期高发!警惕冒充电商物流客服骗局!视频↘</td>\n",
       "      <td>https://so.toutiao.com/search/jump?url=http%3A...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>深入揭批电信网络诈骗套路手法,进一步增强知诈、识诈、防诈、反诈意识,筑牢反诈“防火墙”。 冒...</td>\n",
       "      <td>平安哈尔滨</td>\n",
       "      <td>为全面提升群众防范电信网络诈骗能力和水平，增强安全防范意识，哈尔滨市委政法委认真梳理归纳出目...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>假客服，真诈骗!受害人现身说法!</td>\n",
       "      <td>https://so.toutiao.com/search/jump?url=http%3A...</td>\n",
       "      <td>http://p1-tt.byteimg.com/img/tos-cn-i-qvj2lq49...</td>\n",
       "      <td>1、诈骗分子往往通过不法渠道获取到公民个人信息,再利用192”+、00”开头的境外号码、“区...</td>\n",
       "      <td>潍坊网警</td>\n",
       "      <td>假客服，真诈骗！受害人现身说法！\\r\\n\\r\\n“您好，我是京东客服”\\r\\n\\r\\n“不关...</td>\n",
       "      <td>https://p3-sign.toutiaoimg.com/tos-cn-i-qvj2lq...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>被骗100余万!又是假冒客服诈骗，16人落网</td>\n",
       "      <td>https://so.toutiao.com/search/jump?url=http%3A...</td>\n",
       "      <td>https://p3-sign.toutiaoimg.com/tos-cn-i-tjoges...</td>\n",
       "      <td>电信诈骗案案件回顾近日,牡丹分局北城派出所接到辖区居民陈某报警,称其遭遇冒充客服诈骗,被骗1...</td>\n",
       "      <td>环球网</td>\n",
       "      <td>来源：菏泽公安\\r\\n\\r\\n“双11”刚刚过去\\r\\n\\r\\n“双12”又紧跟着要来了\\r...</td>\n",
       "      <td>https://p3-sign.toutiaoimg.com/tos-cn-i-tjoges...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>警惕!冒充客服类诈骗</td>\n",
       "      <td>https://so.toutiao.com/search/jump?url=http%3A...</td>\n",
       "      <td>https://p3-sign.toutiaoimg.com/tos-cn-i-6w9my0...</td>\n",
       "      <td>冒充客服诈骗主要分为以下几类 01 网购商品出现问题或免费赠送小家电 一是冒充淘宝、京东、抖...</td>\n",
       "      <td>平安石柱</td>\n",
       "      <td>网络购物的快速发展\\r\\n\\r\\n让人们享受到了其带来的便利\\r\\n\\r\\n也产生了一大批网...</td>\n",
       "      <td>https://p3-sign.toutiaoimg.com/tos-cn-i-axegup...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>又出新花招!!“平台客服”骗局它来了</td>\n",
       "      <td>https://so.toutiao.com/search/jump?url=http%3A...</td>\n",
       "      <td>https://p3-sign.toutiaoimg.com/tos-cn-i-tjoges...</td>\n",
       "      <td>接下来给大家分析一下冒充客服诈骗的手法第一步:事主接到电话(固话、手机号码),嫌疑人自称“平...</td>\n",
       "      <td>广州公安</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                       标题                                               标题链接  \\\n",
       "0   近期高发!警惕冒充电商物流客服骗局!视频↘  https://so.toutiao.com/search/jump?url=http%3A...   \n",
       "1        假客服，真诈骗!受害人现身说法!  https://so.toutiao.com/search/jump?url=http%3A...   \n",
       "2  被骗100余万!又是假冒客服诈骗，16人落网  https://so.toutiao.com/search/jump?url=http%3A...   \n",
       "3              警惕!冒充客服类诈骗  https://so.toutiao.com/search/jump?url=http%3A...   \n",
       "4      又出新花招!!“平台客服”骗局它来了  https://so.toutiao.com/search/jump?url=http%3A...   \n",
       "\n",
       "                                             首页缩略封面图  \\\n",
       "0                                                NaN   \n",
       "1  http://p1-tt.byteimg.com/img/tos-cn-i-qvj2lq49...   \n",
       "2  https://p3-sign.toutiaoimg.com/tos-cn-i-tjoges...   \n",
       "3  https://p3-sign.toutiaoimg.com/tos-cn-i-6w9my0...   \n",
       "4  https://p3-sign.toutiaoimg.com/tos-cn-i-tjoges...   \n",
       "\n",
       "                                              首页缩略文段    发布者  \\\n",
       "0  深入揭批电信网络诈骗套路手法,进一步增强知诈、识诈、防诈、反诈意识,筑牢反诈“防火墙”。 冒...  平安哈尔滨   \n",
       "1  1、诈骗分子往往通过不法渠道获取到公民个人信息,再利用192”+、00”开头的境外号码、“区...   潍坊网警   \n",
       "2  电信诈骗案案件回顾近日,牡丹分局北城派出所接到辖区居民陈某报警,称其遭遇冒充客服诈骗,被骗1...    环球网   \n",
       "3  冒充客服诈骗主要分为以下几类 01 网购商品出现问题或免费赠送小家电 一是冒充淘宝、京东、抖...   平安石柱   \n",
       "4  接下来给大家分析一下冒充客服诈骗的手法第一步:事主接到电话(固话、手机号码),嫌疑人自称“平...   广州公安   \n",
       "\n",
       "                                              文章正文内容  \\\n",
       "0  为全面提升群众防范电信网络诈骗能力和水平，增强安全防范意识，哈尔滨市委政法委认真梳理归纳出目...   \n",
       "1  假客服，真诈骗！受害人现身说法！\\r\\n\\r\\n“您好，我是京东客服”\\r\\n\\r\\n“不关...   \n",
       "2  来源：菏泽公安\\r\\n\\r\\n“双11”刚刚过去\\r\\n\\r\\n“双12”又紧跟着要来了\\r...   \n",
       "3  网络购物的快速发展\\r\\n\\r\\n让人们享受到了其带来的便利\\r\\n\\r\\n也产生了一大批网...   \n",
       "4                                                NaN   \n",
       "\n",
       "                                                图片链接  \n",
       "0                                                NaN  \n",
       "1  https://p3-sign.toutiaoimg.com/tos-cn-i-qvj2lq...  \n",
       "2  https://p3-sign.toutiaoimg.com/tos-cn-i-tjoges...  \n",
       "3  https://p3-sign.toutiaoimg.com/tos-cn-i-axegup...  \n",
       "4                                                NaN  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "file = \"./dataset/Fraud_News_Reports/冒充客服诈骗(200).csv\"\n",
    "dataset = pd.read_csv(file)\n",
    "dataset.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 这里封装一个美化长文本的输出函数，将长文本分割成80字换行显示。\n",
    "import json\n",
    "import textwrap\n",
    "\n",
    "def print_json(obj):\n",
    "  print(json.dumps(obj, indent=4, ensure_ascii=False))\n",
    "\n",
    "def pretty_print(text):\n",
    "  wrapped_text = textwrap.fill(text, width=80)  # 设定每行的最大字符数\n",
    "  print(wrapped_text)\n",
    "\n",
    "column_content = \"文章正文内容\"\n",
    "pretty_print(dataset.iloc[1][column_content])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 访问GPT接口有很多二次封装的框架，像著名的langchain，不过这里选用Agently框架\n",
    "from agently import Agently\n",
    "agently = Agently(api_key=\"你的API密钥\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 可以循环读取所有的csv文本中的案例，循环调用GPT接口来为每个案例生成对话集。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "def print_json(obj):\n",
    "  print(json.dumps(obj, indent=4, ensure_ascii=False))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\n",
      "    \"传销诈骗(140).csv\",\n",
      "    \"短信诈骗(200).csv\"\n",
      "]\n"
     ]
    }
   ],
   "source": [
    "# 先看看有哪些类型的案例集文件。\n",
    "import os\n",
    "\n",
    "def get_files(directory, extension='.csv'):\n",
    "  files = [f for f in os.listdir(directory) if f.endswith(extension)]\n",
    "  return files\n",
    "\n",
    "# Example usage:\n",
    "# dataset_path = './dataset/Fraud_News_Reports'\n",
    "dataset_path = './dataset/Fraud_News_Reports_Part'\n",
    "csv_files = get_files(dataset_path)\n",
    "print_json(csv_files)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['“杀猪盘” 诈骗(200).csv',\n",
       " '二手交易诈骗(200).csv',\n",
       " '传销诈骗(140).csv',\n",
       " '兼职刷单诈骗(140).csv',\n",
       " '冒充客服诈骗(200).csv',\n",
       " '微商代理诈骗(200).csv',\n",
       " '投资诈骗(200).csv',\n",
       " '短信诈骗(200).csv',\n",
       " '网络贷款诈骗(162).csv',\n",
       " '网络赌博诈骗(200).csv',\n",
       " '虚假购物诈骗(200).csv',\n",
       " '虚拟货币诈骗(200).csv',\n",
       " '诈骗(150).csv']"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.listdir('./dataset/Fraud_News_Reports')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 封装2个高阶函数\n",
    "# 返回一个能将对话集写入指定文件的write函数，file_name是要写入的文件名。\n",
    "def build_write_func(file_name):\n",
    "  def write(rows, header=False):\n",
    "    df_new = pd.DataFrame(rows, index=range(len(rows)))\n",
    "    df_new.to_csv(file_name, mode='a', header=header, index=False)\n",
    "  return write"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 第2个高阶函数\n",
    "- build_to_rows_func：返回一个将对话集转换为csv rows的函数，统一每条数据的表头和列名。\n",
    "  > case_prefix用于指定案例名称的前缀，这里统一采用文件名作为前缀用于区分不同类型的案例集。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 将对话集转换为行(rows)的函数，case_prefix是案例编号的前缀。\n",
    "def build_to_rows_func(case_prefix=\"\"):\n",
    "  def to_rows(dialog: list, i):\n",
    "    rows = []\n",
    "    for item in dialog:\n",
    "      data = {'case': f'{case_prefix}{i}'}\n",
    "      for k in ['speaker', 'content', 'is_fraud', 'reason']:\n",
    "        data[k] = item[k]\n",
    "      rows.append(data)\n",
    "\n",
    "    return rows\n",
    "\n",
    "  return to_rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'冒充客服诈骗(200)'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 封装一个函数，用于根据文件路径返回文件名。\n",
    "\n",
    "import os\n",
    "def filename(path):\n",
    "  filename_with_ext = os.path.basename(path)\n",
    "  filename, extention = os.path.splitext(filename_with_ext)\n",
    "  return filename\n",
    "\n",
    "filename(file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_dialog(agent, text):\n",
    "  response = agent.load_yaml_prompt(\n",
    "    yaml = prompt_template,\n",
    "    variables={\n",
    "      \"input\": text,\n",
    "      \"language\": 'chinese',\n",
    "    }\n",
    "  ).start()\n",
    "  # 如果遇到markdown格式的json，则尝试用上面封装的remove_markdown_boundary来二次处理.\n",
    "  if isinstance(response, str):\n",
    "    response = json.loads(remove_markdown_boundary(response))\n",
    "  return response['result']\n",
    "\n",
    "print_json(generate_dialog(agent_4o, dataset.iloc[1][column_content]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 开始使用\n",
    "# 主循环，依次为每个文件生成对话集。 --->>> 最终会为每个文件都生成一个对话集。\n",
    "# > excludes用于排除一些文件，适用于目录下有些文件不需要生成对话集的场景。\n",
    "\n",
    "\n",
    "output_dir = \"../dataset/fraud/csv_dialogs\"\n",
    "excludes = [\n",
    "  \"冒充客服诈骗(200).csv\",\n",
    "]\n",
    "os.makedirs(output_dir, exist_ok=True)\n",
    "\n",
    "for fname in csv_files[12:13]:\n",
    "    if fname not in excludes:\n",
    "        input_path = os.path.join(dataset_path, fname)\n",
    "        print(f\"generate for file_path: {input_path} ...\")\n",
    "        generate_dialogs('', input_path, output_dir, 0, 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 最终用于为指定文件生成对话集的函数，传入一个案例报道格式的文件路径，来生成一个对话格式的数据集文件。\n",
    "# 这里的 gentrate_dialog 是调用 agent 生成对话的函数。\n",
    "\n",
    "def generate_dialogs(agent, input_path, output_dir, start_idx=0, end_idx=-1):\n",
    "  dataset = pd.read_csv(input_path)\n",
    "  # print_json(dataset.head(2))\n",
    "  print(f\"dataset length: {len(dataset)}\")\n",
    "  print(f\"generate from {start_idx} to {end_idx} ...\")\n",
    "  # print(f\"111: {dataset[start_idx:end_idx]}\")\n",
    "  # print(f\"222-iterrows(): {dataset[start_idx:end_idx].iterrows()}\")\n",
    "  output_path = f\"{output_dir}/{filename(input_path)}.csv\"\n",
    "  print(f\"output_path: {output_path}\")\n",
    "  write_fn = build_write_func(output_path)\n",
    "  to_rows_fn = build_to_rows_func(filename(input_path))\n",
    "\n",
    "  for i, row in dataset[start_idx:end_idx].iterrows():\n",
    "    # 先临时注释掉，避免重复调用接口 start\n",
    "    # dialog = generate_dialog(agent, row[column_content])\n",
    "    # dialog_rows = to_rows_fn(dialog, i)\n",
    "    # write_fn(dialog_rows, i == 0)\n",
    "    # 先临时注释掉，避免重复调用接口 end 真实使用场景，要放开的\n",
    "    print(f\"**********content: {i + 1}\")\n",
    "    print(f\"**********write dialog count: {i + 1}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "111\n",
      "['传销诈骗(140).csv']\n",
      "generate for file_path: ./dataset/Fraud_News_Reports_Part\\传销诈骗(140).csv ...\n",
      "dataset length: 120\n",
      "generate from 0 to 2 ...\n",
      "111:                                           标题  \\\n",
      "0  涉案2亿的传销骗局:2800元就能当“老板”，“底层”啃土豆白菜供“高层”挥金如土   \n",
      "1                     “深度洗脑”后怕!传销转诈骗，骗局不断在更新   \n",
      "\n",
      "                                                标题链接  \\\n",
      "0  https://so.toutiao.com/search/jump?url=http%3A...   \n",
      "1  https://so.toutiao.com/search/jump?url=http%3A...   \n",
      "\n",
      "                                             首页缩略封面图  \\\n",
      "0  https://p3-sign.toutiaoimg.com/tos-cn-i-tjoges...   \n",
      "1                                                NaN   \n",
      "\n",
      "                                              首页缩略文段     发布者  \\\n",
      "0  打掉了一个特大聚集型传销诈骗犯罪团伙,抓获犯罪嫌疑人13人。该传销在无任何产品销售的情况下,...    红星新闻   \n",
      "1  2023年7月11日17时,新疆奎屯垦区公安局准噶尔路派出所辖区酒香园居民拽着老公赖某向派出...  中国质量新闻   \n",
      "\n",
      "                                              文章正文内容  \\\n",
      "0  花2800元买一套所谓的“化妆品”，就能入会当“老板”；再拉拢新人加入，还能分得“奖金”……...   \n",
      "1  2023年7月11日17时，新疆奎屯垦区公安局准噶尔路派出所辖区酒香园居民拽着老公赖某向派出...   \n",
      "\n",
      "                                                图片链接  \n",
      "0  https://p3-sign.toutiaoimg.com/tos-cn-i-tjoges...  \n",
      "1                                                NaN  \n",
      "222-iterrows(): <generator object DataFrame.iterrows at 0x000002761EAEC8C0>\n",
      "**********write dialog count: 1\n",
      "**********write dialog count: 2\n"
     ]
    }
   ],
   "source": [
    "# output_dir = \"../dataset/fraud/csv_dialogs\"\n",
    "# 因为csv_dialogs 这个目录已经存在生成好的了，所以改个名字\n",
    "output_dir = \"../dataset/fraud/csv_dialogs_new\"\n",
    "\n",
    "excludes = [\n",
    "  \"短信诈骗(200).csv\"\n",
    "]\n",
    "os.makedirs(output_dir, exist_ok=True)\n",
    "# csv_files\n",
    "# csv_files[0:1]\n",
    "print(111)\n",
    "print(csv_files[0:1])\n",
    "for fname in csv_files[0:1]:\n",
    "  if fname not in excludes:\n",
    "    input_path = os.path.join(dataset_path, fname)\n",
    "    print(f\"generate for file_path: {input_path} ...\")\n",
    "    generate_dialogs('', input_path, output_dir, 0, 2)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.5"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "340a546b2c4c9cf5d23eb4a2a4e78e923e0b7afe4d00162258c4442b3ee3b061"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
